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Abstract 


Comparison  of  series  of  live-fire  test  results  with  stochastic  simulation  model 
output  has  been  a  long-standing  problem.  The  usual  approach,  the  order  by 
probability  (OP)  test,  allows  neither  for  quantification  of  typical  adjustments 
made  in  the  implementation  of  the  test  nor  for  sensitivity  analysis.  A  new  test  is 
presented  here  which  may  be  used  in  conjuction  with  the  OP  test  to  provide 
these  missing  features. 
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1.  Vulnerability  Models 


The  task  at  hand  is  to  compare  a  set  of  live-fire  (LF)  results  as  a  group  to  stochastic 
simulation  model  output  in  order  to  determine  if  the  model  adequately  predicts 
the  LF  results.  The  simulation  is  replaced  by  a  simple  probabilistic  model,  and 
statistical  tests  are  conducted  to  determine  if  the  LF  results  are  consistent  with  the 
idealized  simulation  distribution. 

Assume  initially  that  a  stochastic  vulnerability /lethality  (VL)  model  predicts  either 
no  loss  of  function  (LoF)  or  complete  LoF  for  a  particular  component,  system,  or 
vehicle.  Under  a  fixed  set  of  input  conditions,  output  typically  varies  due  to  the 
stochastic  nature  of  the  model.  The  output  of  such  a  model  can  be  represented 
as  X  G  {0,1},  where  Pr[X  =  1]  =  p  =  1  -  Pr[X  =  0]  and  the  single  parameter  p  of 
the  model  lies  in  the  interval  [0,1].  The  values  of  X  have  the  interpretation  that 
X  =  1  indicates  complete  LoF  and  X  =  0  indicates  no  LoF,  so  that  X  is  in  fact 
the  LoF.  This  random  variable  X  has  a  Bernoulli  distribution  with  parameter  p. 
Equivalently,  X  is  a  binomial  random  variable  with  sample  size  1  and  parameter 
p,  denoted  X  ~  B(l,p). 

The  model  independently  exercised  under  k  sets  of  input  conditions  (each 
representing  a  distinct  shot  with  a  particular  target,  threat,  geometry,  etc.)  is 
denoted  by  the  vector  X  =  (X,)f^j  =  (Xi,X2,...,Xfc)  with  vector  parameter  p 
=  iPi)i=v  ^  configuration  i,  and  Pr[X,  =  1]  =  pi 

=  1  -  Pr[Xj  =  0].  The  distribution  of  the  vector  X  is  fc-dimensional  binomial  of 
sample  size  1  and  parameter  p  with  independent  components,  or  X  ~  Bjt(l,p). 
Component  i  of  A:  bias  the  binomial  distribution  X,  ~  B(l,p,  )- 

Each  set  of  conditions  is  associated  with  a  single  experimental  result  in  U  G  {0,1}, 
which  together  constitute  the  data  vector  L  =  G  S  =  {0,1}*^.  So  L  is  a 

fc-vector  of  zeros  and  ones,  and  the  sample  space  has  size  2*^. 

As  VL  models  increase  in  complexity,  they  allow  for  partial  LoFs,  so  X  G  [0, 1].  For 
a  simple  example,  suppose  that  a  vehicle  has  a  maximum  speed  of  60  mph.  If  the 
vehicle  can  maintain  that  speed  after  the  shot,  then  X  =  0  since  there  is  no  LoF.  If 
the  vehicle  cannot  move,  then  there  is  total  LoF  and  X  =  1.  And  if  the  vehicle  can 
operate  at  a  maximum  speed  of  30  mph,  then  one  might  assess  the  LoF  at  X  =  0.5. 

Typically  the  number  of  outputs  for  a  single  input  configuration  is  finite  and  in 
fact  completely  exhausted  after  a  few  hundred  or  a  few  thousand  iterations,  giving 
perhaps  as  many  as  a  dozen  distinct  output  LoFs. 
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In  one  dimension,  the  model  output  LoF,  denoted  X,  can  take  on  any  of  the  n  values 
of  a:  =  with  respective  probabilities  p  =  so  that  Pr[X  =  Xj]  =  pj  where 

Pj  =  1  and  each  p,-  >  0.  The  one-dimensional  model  for  X  is  finite  discrete,  or 
multinomial  with  sample  size  one,  value  parameter  x,  and  probability  parameter 
p.  This  is  denoted  X  ~  M{l,x,p),  and  the  sample  space  or  set  of  possible  values  of 
Xis  {xj}]^i. 

Again,  the  model  can  be  exercised  under  k  sets  of  input  conditions,  each 
representing  a  distinct  shot.  The  VL  model  is  then  A:-dimensional  vector 
multinomial  Mfc(l,x,p)  of  size  1  where  the  value  parameter  list  is  x  =  with 

Xi  =  and  the  probability  parameter  list  is  p  =  {pi)i=i  with  p,  =  (pi,/)”ii  for 

each  I  of  k,  so  that  Pr[X,-  =  Xij]  =  pi^j.  This  means  that  for  shot  i,  the  LoF  value  Xjj 
occurs  in  the  model  with  probability  p,  j.  In  this  case,  it  is  assumed  that  the  data 
vector  of  LF  LoF  assessments  L  =  (Li)-Li  lies  in  the  sample  space  S  = 
which  means  that  each  single  result  L,  lies  in  the  appropriate  set 

A  final  generalization  occurs  when  L  does  not  necessarily  lie  in  S.  For  example, 
it  may  be  reasonable  to  conclude  that  in  one  dimension  the  LF  LoF  assessment 
L  =  0.26  is  consistent  with  the  model  X  ~  Mi;{l,x,p)  where  LoF  values  x 
=  (0.00,0.25,0.50)  have  respective  probabilities  p  =  (0.10,0.85,0.05),  whereas  L  = 
0.99  is  not.  In  this  case,  it  is  desirable  to  test  whether  L  is  close  to  X  in  some  way. 

The  tests  under  consideration  here  assume  that  L  is  an  Mjt(l,x,0)  vector,  perhaps 
Bjt(l,0),  or  close  to  an  Mic{l,x,d)  vector.  Furthermore,  the  VL  model  is  assumed 
to  be  X  ~  Mjt(l,x,p).  These  tests  all  take  the  form  of  either 

Ho  :  e  =  p 

Hi ;  0  /  p  (1) 


or 


Ho:EL  =  EX 

Hi:  EL  EX.  (2) 

The  model  parameter  (x,p)  is  estimated  from  n  iterations  of  the  VL  model,  so  the 
output  is  M.k{n,x,p).  It  is  thus  possible  to  provide  error  bounds,  etc.,  for  the 
estimates  of  (x,p).  This  estimation  problem  is  ignored  without  further  mention, 
and  (x,p)  is  taken  to  be  the  true  parameter  of  the  VL  model. 

More  detailed  information  about  the  concepts  of  probability  and  statistics  used 
here  are  presented  in  the  Appendix. 
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2.  The  Order  by  Probability  (OP)  Test 


The  OP  test  was  originally  designed  to  check  the  consistency  of  component-wise  or 
system-wise  VL  model  predictions  for  a  single  threat-target  interaction,  where  each 
vector  dimejision  represents  a  single  component  or  system  in  the  vehicle.^  Each 
of  k  components  survives  or  is  destroyed  independently,  so  the  appropriate  model 
for  a  single  shot  against  a  single  vehicle  is  vector  binomial  Bjt(l,  0).  The  parameter 
vector  is  e  =  in  which  0,  represents  the  probability  that  the  i*  component 

is  killed.  A  modification  of  the  original  test  exploits  the  empirical  distribution 
of  the  simulation  vector  for  use  when  the  component  or  system  kill  probabilities 
are  deemed  nonindependent.  A  reliable  estimate  of  the  empirical  simulation 
distribution  may  be  difficult  to  obtain.  For  example,  a  set  of  20  components 
presents  over  one  million  possible  outcomes;  furthermore,  a  large  sample  may  be 
required  to  accurately  characterize  the  relative  frequency  of  occurrence  of  the  less 
likely  outcomes. 

However,  distinct  threat-target  interactions  (separate  shots)  are  independent,  so 
the  binomial/multinomial  OP  test  is  applicable  to  a  set  of  separate-shot  complete- 
vehicle  LoFs.  The  original  formulation  of  the  OP  test  applies  when  the  response 
and  predictions  are  in  {0,1}  for  each  dimension.  The  test  can  be  extended  to  the 
case  of  the  Mkil,x,e)  version  of  the  VL  model  discussed  in  the  previous  section, 
where  vehicle  LoFs  lie  in  [0,1].  The  i*  threat-target  interaction  of  k  assumes 
each  of  its  n,  possible  LoF  values  with  respective  probability  pi,j,  and  the 

interactions  are  independent.  So  the  simulation  distribution  is  taken  to  be  vector 
multinomial  of  sample  size  1. 

A  technical  description  of  the  OP  test  follows,  first  in  the  original  binomial  form 
and  then  in  the  multinomial  form.  This  level  of  representation  engenders  formulae 
that  easily  translate  into  computational  instructions. 

2.1  Technical  Description 

In  the  binomial  case  the  simulation  distribution  is  X  ~  Bjt(l,p),  and  the  experiment 
distribution  is  assumed  to  be  L  ~  B)t(l,0).  The  OP  hypothesis  test  is 

Ho  :  0  =  p 

Hi:0#p.  (3) 

^Webb,  D.  W.  'Tests  for  Consistency  of  Vulnerability  Models."  BRL-TR-3030,  U.S.  Army  Ballistic 
Research  Laboratory,  Aberdeen  Proving  Ground,  MD,  August  1989. 
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Since  E  Z  =  0  when  Z  ~  this  is  the  same  as 


Ho:EL  =  EX 

Hi:EL^EX.  (4) 

The  OP  test  is  a  generalized  likelihood  ratio  test.  The  sample  distribution  is  Z 
~  Bjt(l,0),  where  the  parameter  space  is  0  =  [0,1]^  and  the  null  space  is  0o  =  {p}- 
The  test  of 


Ho  :  0  e  ©0 

Hi:0€0-0o  (5) 


uses  the  generalized  likelihood  ratio  A(L)  as  a  test  statistic.  The  generalized 
likelihood  ratio  function  is 


_  s^Pee0oPre[Z  =  t] 

=  t] 


=  Prp[Z  =  t]  =  /p(t), 


(6) 


where  fp  is  given  by  equation  (A-9)  in  the  Appendix,  so  the  test  statistic  is 
A(Z)  =  fp{Z).  Construction  of  the  test  statistic  distribution  follows.  Let 
be  an  enumeration  of  the  sample  space  S  =  {0,1}*^  so  that  fp{Z)  takes  the  values 
{fp  (t«  ) .  Then  the  distribution  of  /p  ( Z )  is  given  by 


Pr,l/p(Z)  =/,(())  =  E  Pr,|Z  =  r»|  = 

/p{Tn)=/p(0  /p(T„)=/p(t) 


where,  again,  fe  is  as  in  equation  (A-9).  Now,  /p(Z)  has  small  values  significant. 
The  order  can  be  reversed  by  using  for  a  test  statistic  either  T{Z)  =  gp(Z) 
=  1  -/p(Z),  or,  equivalently,  the  rank  T{Z)=gp{Z)  of  gp{Z)  in  {gp(T„)}^Li,  both 
of  which  have  large  values  significant.  Note  that  gp  —  fp,  the  descending  rank  of 

/p(Z)in  {fpirn)}lU. 

In  either  case,  the  cdf  of  T(Z)  is 

Pe{t)  =  Xl/e('rn)f[r(T„),co)(t)/  i-e., 

W  =  1 
2*^ 

«=i 

2‘ 

n=l 
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and  the  null  distribution  is  given  when  6  =  p.  The  decision  rule  is  to  reject  if 
T(Z)  >  Ta,  where  the  critical  value  is  the  usual  T«  =  Qp(l  —  ao).  The  power  of  the 
test  under  the  alternative  6  is  1  -  =  1  -  fe  ( Ta). 

Extension  to  the  multinomial  VL  model  is  transparent.  In  this  case,  the  sample 
distribution  is  Z  ~  0).  The  hypothesis  test  again  takes  the  form 

Ho  :  0  =  p 

Hx-.e^p,  (9) 

where  the  simulation  distribution  is  X  ~  Mk(l,x,p)  and  the  experiment 
distribution  is  L  ~  0).  Note,  the  values  x  that  the  model  and  data  take  are 

fixed  and  assumed  to  be  identical.  It  is  the  probability  of  occurrence  0  of  these 
values  that  is  being  tested.  In  terms  of  components,  the  parameter  of  interest 
is  0  =  ((0i,;)”ii)iLi.  In  the  multinomial  case,  the  test  is  not  a  test  of  EL  =  EX, 
since  EZ  =  for  Z  ~  At  any  rate,  this  is  another 

generalized  likelihood  ratio  test,  and  construction  of  the  test  statistic  is  analogous 
to  the  binomial  case.  The  generedized  likelihood  ratio  is 


suPe60oPrM[^  =  ^] 

SUp0g0Pr:c,0[Z  =  t] 


—  Prar,p['Z  —  f]  — 


(10) 


where  fx,p  is  given  by  equation  (A-13),  so  the  test  statistic  is  fx,p  ( Z ) .  Let  {t„  be 
an  enumeration  of  the  sample  space  S  =  where  N  =  n/=i  ”!•  Then 

fx,p ( Z )  is  defined  on  {fx,p{^n ) •  The  distribution  of  fx,p  ( Z )  is  given  by 

Pra;,e[/*,p(Z)  =  fx,p{i)]  =  ^  Prx,0[Z  =  t„] 

fx,pM=fx,p(t) 

~  y  ^  fx.si'^n)/  (11) 

fx,p(^n)=fx,p{t) 

where  fx,e  is  given  by  equation  (A-13).  It  is  equivalent  to  use  either  T(Z) 
=  gx,p  ( z )  =  1  -  ( Z )  or  the  rank  T  ( Z )  =  ( Z )  of  gx,p  ( Z )  in  {gx,p  (t„  )  ,  both 
of  which  have  large  values  significant.  Again,  ^  =  fx^j,. 


5 


In  either  case,  the  cdf  of  T(  Z )  is 


N 

Px,e(t)  = 

n=l 

N 

n=l 

(12) 

n=l 

and  the  null  distribution  is  given  when  d  =  p.  The  decision  rule  is  to  reject  if 
T(Z)  >  Ta,  where  the  critical  value  is  =  Qx,r{l  -  «o).  The  power  of  the  test 
under  the  alternative  (x:,0)  is  1  -  ^  =  1  -  Fx,e{Ta)- 

In  practice  there  may  be  "degenerate"  distributions,  which  arise  when  there  is  a 
single  number  coming  out  of  X  in  some  component.  That  is  to  say  «,  =  1,  X,  =  Xj, 
and  Pi  =  1  for  some  i.  If  Li  7^  Xi,  reject  Hq.  Otherwise,  test  the  nondegenerate 
components  as  above. 

Of  course,  the  multinomial  model  subsumes  the  binomial  model.  With  model 
output  and  data  limited  to  {0,1}  it  is  certain  that  L  lies  in  S,  the  sample  space  of 
the  model  X.  This  is  not  necessarily  so  for  the  multinomial  model.  For  some  i 
it  may  be  that  Li  does  not  lie  in  {xi,j}^j‘^i.  This  event  occurs  with  null  probability 
zero,  hence  ranks  higher  than  any  vector  the  model  produces.  And  so  the  null 
hypothesis  is  rejected  here. 

2.2  The  Need  for  Sensitivity 

Analysts  have  attempted  to  modify  the  OP  test  in  order  to  eliminate  the  strict 
requirement  that  L  €  S.  Such  modifications  usually  invoke  some  comparison 
of  Li  with  model  output  for  example,  through  using  histogram  techniques 
or  "binning  close  values  together,"  which  implies  the  construction  of  a  distance 
measure  on  a  space  containing  the  model  output  X  and  data  vector  L.  In  other 
words,  this  introduces  into  the  analysis  a  concept  of  distance  between  model  LoF 
vectors.  The  OP  test  does  not  use  the  model  output  values,  only  their  probabilities 
of  occurrence.  Model  prediction  values  are  used  as  an  index  to  the  probabilities 
that  are  being  compared.  So  the  actual  data  and  model  values  have  no  part  in  the 
OP  test  and  are  only  introduced  in  an  effort  to  generalize  the  test  and  as  an  aid  in 
understanding  OP  test  results.  The  OP  test  statistic  cannot  provide  any  indication 
of  the  effect  of  these  adjustments.  However,  it  is  possible  to  construct  a  test  that 
incorporates  model  outcome  values  (in  addition,  of  course,  to  their  probabilities  of 
occurrence)  and  enables  quantification  of  such  effects. 
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A  similar  difficulty  can  arise  in  the  application  of  the  OP  test  even  when  L 
e  S.  Suppose  the  test  results  in  rejection.  The  analyst  may  wonder  why  and 
ask  if  a  particular  component  is  responsible  for  the  reject  decision.  The  usual 
tack  taken  here  is  to  examine  the  location  of  L  among  the  values  of  X  to  gain 
insight  about  the  behavior  of  g^,p(L)  among  g^,p(X).  Again,  the  OP  test  can 
only  provide  information  when  the  real  (or  altered)  experiment  results  lie  in  the 
simulation  sample  space.  The  new  test  to  be  proposed  can  accommodate  arbitrary 
incremental  adjustments  to  the  experiment  data  and  allow  for  true  sensitivity 
analysis. 

This  situation  also  arises  in  power  consideration.  A  significance  level  a  may  be 
chosen  to  manipulate  p  against  a  certain  alternative,  seeking  a  low  jS  against  a 
"highly  unlikely"  outcome.  The  unlikeliness  of  an  outcome  is  most  likely  to  be 
judged  by  its  distance  from  likely  outcomes  in  terms  of  value  and  not  probability. 

Sometimes  the  OP  test  gives  surprising  results.  Consider  a  multinomial  example 
in  one  dimension  with  x  =  (0,0.15,0.9)  and  p  =  (0.8,0.04,0.16).  Statistics  are  g^p 
=  (1,3,2)  with  cdf  values  (0.8,1,0.96).  Set  the  significance  level  at  Uo  —  0.16.  Then 
Ttt  =  2  and  the  reject  region  is  gx,p  e  {3},  or  x  e  {0.15}.  If  the  data  is  L  =  0.15,  the 
test  rejects  Hq.  Note  that  the  expected  value  of  the  model  is  E  X  =  0  x  0.8  +  0.15 
X  0.04+ 0.9  X  0.16  =  0.15.  So  the  OP  test  asserts  that  the  data  does  not  come  from 
the  model  distribution  when  the  data  is  in  fact  the  mean  of  the  model  distribution 
and  is  in  the  model  sample  space.  Note  that  if  the  data  is  L  =  0.9,  the  OP  test  does 
not  reject  Hq.  This  value  of  x  is  far  removed  from  the  mean  of  the  distribution  in 
the  following  sense.  Consider  a  statistic  that  is  the  rank  of  distance  from  the  mean. 
That  is,  T*  =  (2,1,3)  with  cdf  values  (0.84,0.04,1).  Mtha  =  0.16,  the  test  based  on 
this  statistic  has  T^  =  2  and  so  rejects  Hq  when  L  =  0.9  but  not  when  L  =  0.15.  On 
one  hand,  L  =  0.15  is  a  rare  event  and  L  =  0.9  is  more  likely  to  occur.  On  the  other 
hand,  L  =  0.15  lies  nearer  to  a  group  of  more  likely  outcomes  than  does  L  =  0.9. 

It  is  beneficial  to  quantify  the  concept  of  distance  that  is  already  being  used  in 
conjunction  with  OP  test  analysis  of  LF  experiments.  This  can  be  accomplished 
with  the  use  of  a  statistic  and  hypothesis  test  based  on  a  measure  of  central 
tendency  in  the  model/simulation  output  value  space.  Such  a  test  is  now 
proposed. 


3.  The  MX  Test 


The  VL  model  with  output  X  in  the  sample  space  S  =  is 

fc-dimensional  vector  multinomial  Mjt(l,x,p)  with  value  parameter  list  x  and 
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probability  parameter  list  p.  A  test  of  the  hypothesis 


Ho:EL  =  EX 

Hi:  EL  7^  EX  (13) 


is  conducted  using  the  test  statistic  T  =  Mx,p  given  by 


where 


(Z,-EX,f 

VarX, 


m,-  =  E  X,- ,  sj  =  Var  X; ,  and  Ai  (f ) 


t-nii 

Si 


(14) 


This  is  a  test  of  E  L  =  E  X  given  that  Var  L  =  Var  X  under  Hq.  It  is  not  required  that 
L  e  S.  The  distribution  of  Mx,p  is,  of  course, 


F(f)  =  Pr[M,,p(Z)<t]. 


(15) 


When  Z  ~  Mjt(l,2,0),  this  is 


N 

hAt)  =  (16) 

n=l 

where  Z  6  =  {t„}^^i  and  N  =  OLl”!'  f^-e  given  by 

equation  (A-13).  Note  that  the  x„  comprise  an  enumeration  of  the  Z  sample  space, 
not  the  sample  space  of  the  simulation  X.  The  distribution  parameter  {x,p)  of 
X  enters  the  calculation  through  the  indicator  of  [Mx,pi'^n),°°)>  and  the  probability 
that  Mx,p(Z)  assumes  a  particular  value  is 

Pr,,e[M,,p(Z)  =  M,,p(t„)]  =  J];Pr,,<,[Z  =  t,] 

1 

=  y  Az,si'^i)  iAlx,p{Xi)  =  M.x,p{Xn)} ,  (17) 

I 

where  7  =  {z  |  Mx,p(T/)  =  Mx,p(T«)},  which  is  a  sum  of  probabilities  from  the 
distribution  of  Z.  The  null  distribution  of  M*,p(Z)  is  given  when  Z  ^  Mk{l,x,p). 

The  statistic  M.x,p  has  an  approximate  chi-square  distribution,  Mx,p  ~  xl'  under 
the  null  hypothesis.  The  distribution  of  Mx,p  under  an  alternative  hypothesis 
is  approximately  noncentral  chi-square,  Mx,p  Xk,5'  with  the  usual  noncentrality 

parameter  S  =  2f=i  (E  Z,-  -  E  X,)^/ Var  X,-. 

This  test,  based  on  a  multinomial  distribution  with  a  x^  approximation,  will  be 
referred  to  as  the  MX  test. 
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4.  Demonstration 


For  illustrative  purposes,  consider  a  test  for  the  binomial  model  with  parameter 
p  =  (0.1,0.2,0.3,0.4,0.8)  at  the  significance  level  ao  =  0.1.  Power  is  calculated 
against  the  specific  alternative  6  =  (0.9, 0.2, 0.3, 0.6, 0.8).  Distributions  of  the  test 
statistics  are  shown  under  the  null  and  alternative  hypotheses,  along  with  a, 
power,  and  values  of  the  test  statistics.  Figure  1  shows  the  MX  test  statistic  Mx,j) 
with  its  approximation.  Figure  2  shows  the  rank  MJ  p  of  the  MX  statistic. 
Figure  3  shows  the  OP  test  statistic  gx,p.  Figure  4  shows  the  rank  gj  p  of  the 
OP  statistic.  Correspondence  of  the  null  ranks  is  shown  in  Figure  5,  along  with 
the  sizes  and  probabilities  of  the  reject  and  accept  regions.  True  significance 
levels  are  0.025  +  0.066  =  0.091  for  the  OP  test  and  0.066  +  0.018  =  0.084  for 
the  MX  test.  Note  that  Pr  [reject  MX  |  reject  OP]  =  0.066/0.091  =  0.73  and  that 
Pr[reject  OP  |  reject  MX]  =  0.066/0.084  =  0.79.  So,  these  two  different  tests  do  not 
always  yield  the  same  decision.  The  probability  that  a  random  point  passes  one 
test  and  fails  the  other  is  0.025  +  0.018  =  0.043,  and  3  points  out  of  32  lie  in  this 
region. 

To  assess  the  general  power  characteristics  of  these  tests  consider,  for  fixed 
dimension  k,  specifying  a  null  distribution  (pi)fLi  with  uniform  random 
components  and  testing  this  against  a  uniform  random  alternative  for  fixed 

ao.  This  gives  a  random  value  for  the  Type  II  error  probability  p.  Repeat  this  2500 
times  to  get  empirical  distributions,  which  are  estimates  of  the  distributions  of  p 
for  a  test  of  a  random  null  against  a  random  alternative.  Estimated  median  power 
1—pis  shown  for  various  values  of  k  in  Table  1.  The  power  of  the  OP  is  greater  by 
as  much  as  2%,  so  there  is  little  appreciable  difference  between  the  two  tests. 


5.  Application 


Data  from  a  series  of  14  LF  tests  will  now  be  used  to  illustrate  these  statistical 
procedures  by  comparing  the  LF  results  to  predictions  generated  by  the 
MUVES/S2  simulation  model.  The  LF  LoF  assessment  data  L  and  parameters 
for  the  LoF  simulation  distribution  X  are  presented  in  Table  2  for  MLoF  (mobility 
LoF)  and  Table  3  for  FLoF  (firepower  LoF).  The  parameters  of  the  simulation  model 
are  LoF  values  x  and  their  respective  probabilities  of  occurrence  p.  Tables  4  and  5 
include  respectively  for  MLoF  and  FLoF  the  additional  quantities  involved  in  the 
calculation  of  the  MX  statistic  detailed  in  equation  (14). 
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First,  consider  the  MLoF  data.  The  five  degenerate  (single-valued)  distributions  of 
shots  3,  8,  9,  10,  and  13  agree  with  their  LF  results,  so  we  proceed  to  analyze  the 
remaining  set  of  size  9.  Note  that  L4,  Lg,  and  L12  do  not  lie  in  the  sample  space  of 
the  simulation  distribution.  So  the  OP  test  is  not  applicable  here. 

However,  one  may  argue  that,  in  practice,  the  L4  value  of  0.26  is  "close  enough"  to 
the  value  of  0.22  in  the  model  distribution  for  that  shot  for  the  two  to  be  considered 
equal.  This  justifies  changing  the  value  of  L4  to  0.22  for  the  purposes  of  conducting 
the  OP  test.  Likewise,  Le  =  1  and  X6,4  =  0.96  are  "close",  so  we  change  Le  to  0.96. 
Then,  noting  that  Ln  =  0.04  is  close  to  ^12,2  =  0.02,  we  declare  that  Lu  =  0.02.  The 
modified  LF  vector  now  lies  in  the  simulation  sample  space,  and  the  OP  test  can  be 
applied.  The  resulting  p-value  is  0.19,  insufficient  evidence  to  reject  discrepancy 
at  the  10%  significance  level. 

On  the  other  hand,  L12  =  0.04  is  equally  close  to  Xi2,3  =  0.06.  So  we  might  as  well 
declare  that  L12  =  0.06.  The  OP  test  in  this  case  has  a  p-value  of  0.086,  indicating 
disagreement  between  the  model  and  the  experiment  at  the  10%  level. 

So,  for  this  particular  example,  the  OP  test  outcome  can  be  manipulated  at  will  to 
conclude  consistency  or  inconsistency.  Also,  the  effects  of  the  adjustments  made 
in  this  process  cannot  be  quantified. 

Now  consider  application  of  the  MX  test  to  the  MLoF  data.  Even  though  the  LF 
results  from  shots  4,  6,  and  12  do  not  lie  in  the  simulation  distribution  they  are 
considered  "close  enough"  to  justify  use  of  the  MX  test.  Unlike  the  application  of 
the  OP  test  in  dealing  with  this  discrepancy,  no  alteration  of  the  data  is  necesary 
for  the  MX  test. 

The  p-value  for  the  exact  MX  test  is  0.069,  and  the  p-value  of  the  approximation 
to  the  MX  distribution  is  0.12.  These  levels  most  likely  but  marginally  indicate  that 
Ho  should  be  rejected. 

Shot  L4  has  the  greatest  \A\  of  3,  so  this  may  be  the  shot  most  responsible  for  the 
rejection.  The  data  here  is  L4  =  0.26,  to  be  compared  with  X4  parameters  of  X4 
=  (0.02,0.06,0.1,0.22)  andp4  =  (0.36,0.38,0.13,0.13),  which  give  a  mean  of  0.0716 
and  a  standard  deviation  of  0.0631.  Deleting  L4  from  the  data  set  and  performing 
the  MX  test  on  the  remaining  set  of  size  8  yields  an  exact  p-value  of  0.78,  and  an 
approximate  p-value  of  0.74. 

So  the  MX  test  has  been  used  here  to  isolate  the  single  shot  that  is  responsible  for 
possible  discrepancy  between  the  simulation  predictions  and  LF  results. 

In  the  maimer  of  a  sensitivity  analysis,  the  effects  on  the  p-value  of  altering  the  data 
values  L4,  Lg,  and  L12  are  shown  in  Table  6.  The  other  six  nondegenerate  shots 
retain  their  original  data  values,  which  did  not  need  to  be  altered  for  application 
of  the  OP  test;  and  sensitivity  analysis  is  conducted  on  the  full  nondegenerate  data 
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set  of  nine  shots.  The  MX  test  enables  quantification  of  the  effects  of  altering  data, 
which  is  not  possible  relying  solely  on  the  OP  test.  This  is  certainly  desirable. 
Clearly,  the  value  of  L4  is  the  main  factor  in  the  outcome  of  the  test.  Given  that 
Li2  =  0.04  is  close  enough  to  0.02  or  0.06,  and  that  Le  =  1  is  close  enough  to  0.96, 
we  see  that  changing  L4  from  0.26  to  0.24  results  in  a  determination  of  model 
consistency  at  the  10%  level,  independent  of  the  value  of  L12  e  {0.02,0.04,0.06} 
or  of  Le  G  {0.96,1}.  Therefore,  given  that  L4  =  0.26  is  close  enough  to  0.24,  we 
conclude  that  the  MLoF  model  and  LF  results  are  consistent. 

Similarly,  changing  the  value  of  L4  from  0.26  to  0.22  raises  the  exact  p-value  to 
0.22.  A  reasonable  conclusion  to  draw  from  this  exercise  is  that  marginal  rejection 
is  solely  caused  by  underestimation  on  the  part  of  the  simulation  by  an  amount  in 
the  neighborhood  of  0.02-0.04  for  shot  4. 

Therefore,  the  decision  to  accept  or  reject  model  consistency  hinges  on  the  practical 
significance  of  this  quantifiable  discrepancy. 

A  similar  consideration  applies  to  the  FLoF  data.  The  p-values  are  essentially  zero, 
and  this  can  be  attributed  to  shots  5  and  6  in  which  L5  and  Ls  are  both  equal  to  0.05. 
As  can  be  seen  in  Table  7,  reduction  of  these  data  to  somewhere  between  0.02  and 
0.03  results  in  p-values  exceeding  0.2.  The  test  is  highly  sensitive  to  the  values 
of  L5  and  Le,  and  the  original  conclusion  of  inconsistency  results  from  the  model 
underestimating  the  LF  responses  of  0.05  for  shots  L5  and  Lg  by  an  amount  in  the 
neighborhood  of  0.02-0.03. 

Again,  the  MX  test  has  allowed  for  quantification  of  the  modeling  discrepancy 
responsible  for  rejection  of  consistency,  which  then  must  be  interpreted  in  the  light 
of  practical  significance. 

In  conclusion,  the  MX  test  can  be  used  to  perform  sensitivity  analyses  in  tests  for 
the  consistency  of  live-fire  results  and  simulation  predictions.  The  MX  test  can  be 
used  to  identify  the  critical  shots  that  cause  decisions  of  inconsistency  for  a  series 
of  predictions.  The  MX  test  can  also  be  used  to  quantify  the  inconsistency  in  those 
predictions  in  a  manner  not  possible  with  the  OP  test. 
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n=16 

p=0.066 


* 


n=13 

p=0.891 


n=1 

p=0.018 


Table  1.  Demonstration  median  power  against  random  alternative. 


k  OP  MX 

5  0.62  0.61 

6  0.69  0.69 

7  0.76  0.74 

8  0.78  0.77 

9  0.84  0.82 

10  0.85  0.84 
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Table  2.  Basic  MLoF  data  and  model. 


i 

li 

ni 

i 

1 

2 

3 

4 

5 

1 

0 

3 

0 

0.02 

0.06 

p 

0.88 

0.1 

0.02 

2 

1 

2 

X 

0 

1 

V 

0.32 

0.68 

3 

1 

1 

X 

1 

V 

1 

4 

0.26 

4 

X 

0.02 

0.06 

0.1 

0.22 

V 

0.36 

0.38 

0.13 

0.13 

5 

1 

2 

X 

0 

1 

V 

0.56 

0.44 

6 

1 

4 

X 

0 

0.02 

0.8 

0.96 

V 

0.21 

0.29 

0.46 

0.04 

7 

1 

2 

X 

0 

1 

0.34 

0.66 

8 

0 

1 

X 

0 

1 

9 

1 

1 

X 

1 

1 

10 

0 

1 

X 

0 

1 

11 

1 

2 

X 

0 

1 

0.3 

0.7 

12 

0.04 

5 

X 

0 

0.02 

0.06 

0.1 

0.26 

0.27 

0.42 

0.16 

0.14 

0.01 

13 

1 

1 

X 

1 

1 

14 

1 

2 

X 

0 

1 

V 

0.18 

0.82 
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Table  3.  Basic  FLoF  data  and  model. 


i  Li  n;  j 


1 

2 

3 

4 

5 

6 

7 

8 

1 

0.98 

2 

X 

0 

0.98 

p 

0.06 

0.94 

2 

0.02 

3 

X 

0 

0.02 

0.04 

p 

032 

0.66 

0.02 

3 

1 

1 

X 

1 

p 

1 

4 

0.94 

3 

X 

0.06 

0.56 

0.99 

p 

0.38 

0.5 

0.12 

5 

0.05 

2 

X 

0 

0.02 

p 

0.56 

0.44 

6 

0.05 

2 

X 

0 

0.02 

p 

0.51 

0.49 

7 

0.02 

2 

X 

0 

0.02 

p_ 

0.34 

0.66 

8 

0 

1 

X 

0 

p_ 

1 

9 

0.02 

1 

X 

0.02 

p_ 

1 

10 

0 

1 

X 

0 

p_ 

1 

11 

1 

2 

X 

0 

1 

p_ 

0.3 

0.7 

12 

1 

8 

X 

0 

0.06 

0.14 

0.2 

0.56 

0.66 

0.98 

1 

0.12 

0.17 

0.06 

0.02 

0.05 

0.01 

0.43 

0.14 

13 

1 

1 

X 

1 

L. 

1 

1  2x01 

p  0.18  0.82 
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Table  4.  MLoF  data  and  model. 


i 

Li 

Si 

i 

1 

2 

3 

4  5 

1 

0 

0.0032 

0.0101 

-0.32 

3 

X 

0 

0.02 

0.06 

V 

0.88 

0.1 

0.02 

2 

1 

0.68 

0.466 

0.69 

2 

X 

0 

1 

p 

0.32 

0.68 

3 

1 

1 

0 

0 

1 

X 

1 

p 

1 

4 

0.26 

0.0716 

0.0631 

3 

4 

X 

0.02 

0.06 

0.1 

0.22 

p 

0.36 

0,38 

0.13 

0.13 

5 

1 

0.44 

0.496 

1.1 

2 

X 

0 

1 

p 

0.56 

0.44 

6 

1 

0.412 

0.402 

1.5 

4 

X 

0 

0.02 

0.8 

0.96 

p 

0.21 

0.29 

0.46 

0.04 

7 

1 

0.66 

0.474 

0.72 

2 

X 

0 

1 

p 

0.34 

0.66 

8 

0 

0 

0 

0 

1 

X 

0 

£_ 

1 

9 

1 

1 

0 

0 

1 

X 

1 

1 

10 

0 

0 

0 

0 

1 

X 

0 

1 

11 

1 

0.7 

0.458 

0.65 

2 

X 

0 

1 

P_ 

0.3 

0.7 

12 

0.04 

0.0346 

0.0403 

0.13 

5 

X 

0 

0.02 

0.06 

0.1  0.26 

0.27 

0.42 

0.16 

0.14  0.01 

13 

1 

1 

0 

0 

1 

X 

1 

p_ 

1 

14 

1 

0.82 

0.384 

0.47 

2 

X 

0 

1 

p 

0.18 

0,82 
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Table  5.  FLoF  data  and  model. 


i  Lj  mi  Si  Ai  tii  j 

12345678 


1  0.98 

0.921 

0.233 

0.25 

2 

X 

0 

0.98 

0.06 

0.94 

2  0.02 

0.014 

0.01 

0.60 

3 

X 

0 

0.02  0.04 

0.32 

0.66  0.02 

3  1 

1 

0 

0 

1 

a: 

1 

L 

1 

4  0.94 

0.422 

0.313 

1.70 

3 

0.06 

0.56  0.99 

V_ 

0.38 

0.5  0.12 

5  0.05 

0.0088 

0.00993 

4.10 

2 

X 

0 

0.02 

P_ 

0.56 

0.44 

6  0.05 

0.0098 

0.01 

4.00 

2 

X 

0 

0.02 

V_ 

0.51 

0,49 

7  0.02 

0.0132  0.00947  0.72 

2 

X 

0 

0.02 

V_ 

0.34 

0.66 

8  0 

0 

0 

0 

1 

X 

0 

1 

9  0.02 

0.02 

0 

0 

1 

X 

0,02 

L 

1 

10  0 

0 

0 

0 

1 

X 

0 

V_ 

1 

11  1 

0.7 

0.458 

0.65 

2 

X 

0 

1 

V_ 

0.3 

0.7 

12  1 

0.619 

0.439 

0.87 

8 

X 

0 

0.06  0.14  0.2 

0.56  0.66  0.98  1 

V_ 

0.12 

0.17  0.06  0.02 

0.05  0,01  0.43  0.14 

13  1 

1 

0 

0 

1 

X 

1 

V_ 

1 

14  1 

0.82 

0.384 

0.47 

2 

X 

0 

1 

Tp  0.18  0.82 


21 


Table  6.  MLoF  p-value  sensitivity. 


u 

u 

Li2 

p-value 

MX 

OP 

note 

0.22 

0.96 

0.02 

0.23 

0.19 

pass  OP 

0.22 

0.96 

0.04* 

0.24 

- 

0.22 

0.96 

0.06 

0.21 

0.086 

fail  OP 

0.22 

1* 

0.02 

0.21 

- 

0.22 

1* 

0.04* 

0.22 

- 

0.22 

1* 

0.06 

0.19 

- 

0.24 

0.96 

0.02 

0.14 

- 

0.24 

0.96 

0.04* 

0.14 

- 

0.24 

0.96 

0.06 

0.12 

- 

0.24 

1* 

0.02 

0.12 

- 

0.24 

1* 

0.04* 

0.13 

- 

0.24 

1* 

0.06 

0.11 

- 

0.26* 

0.96 

0.02 

0.074 

- 

0.26* 

0.96 

0.04* 

0.076 

- 

0.26* 

0.96 

0.06 

0.067 

- 

0.26* 

1* 

0.02 

0.067 

- 

0.26* 

1* 

0.04* 

0.069 

- 

true  data  vector 

0.26* 

1* 

0.06 

0.063 

- 

"*"  denotes  actual  data  value 

Table  7.  FLoF  p-value  sensitivity. 


u 

Is 

U 

p-value 

MX 

OP 

note 

0.94* 

0.02 

0.02 

0.56 

- 

0.94* 

0.02 

0.03 

0.23 

- 

0.94* 

0.02 

0.05* 

0.037 

- 

0.94* 

0.03 

0.02 

0.21 

- 

0.94* 

0.03 

0.03 

0.087 

- 

0.94* 

0.03 

0.05* 

0.012 

0.94* 

0.05* 

0.02 

0.028 

- 

0.94* 

0.05* 

0.03 

0.0091 

0.94* 

0.05* 

0.05* 

0.0000002 

- 

true  data  vector 

0.99 

0.02 

0.02 

0.47 

0.339 

0.99 

0.02 

0.03 

0.19 

- 

0.99 

0.02 

0.05* 

0.031 

- 

0.99 

0.03 

0.02 

0.18 

- 

0.99 

0.03 

0.03 

0.078 

- 

0.99 

0.03 

0.05* 

0.0092 

0.99 

0.05* 

0.02 

0.024 

- 

0.99 

0.05* 

0.03 

0.0073 

- 

0.99 

0.05* 

0.05* 

0 

- 

"*"  denotes  actual  data  value 

Intentionally  Left  Blank. 


Appendix.  Background,  Notation,  and  Terminology 


The  material  in  this  section  can  be  found  in  any  standard  text  on  mathematical 
statistics,  such  as  Mood,  Graybill,  and  Boes.^  It  is  presented  here  for  easy  reference. 


A.l  Some  Useful  Functions 

The  indicator  function  for  the  set  S  is 


h{t)  = 


1,  teS 

0,  t^S. 


The  cumulative  distribution  function  (cdf)  of  a  univariate  random  variable  (rv)  X 
is  the  left-continuous  function 

F(f)  =  Pr[X<f],  teR  (A-2) 

and  its  right-continuous  pseudo-inverse  is  the  quantile  function  (qf) 

Q(u)  =  inf{f:F(f)^M},  w€[0,l].  (A-3) 

For  continuous  distributions  Q  =  F~^.  For  any  rv,  Q(0)  =  — <». 

A  discrete  finite  random  variable  X  defined  on  the  sample  space  S  =  has 

probability  function  f{t)  =  Pr[X  =  f]  and  cdf 

n  n 

F{t)  =  Pr[X  ^  f]  =  ^Pr[X  =  .  (A-4) 

1=1  i=l 

The  empirical  cdf  based  on  the  i.i.d.  data  {X,  }”^j  is  the  discrete  cdf  of  n 
equiprobable  points 

1=1 

and  its  inverse  is  the  empirical  qf  Qniu).  For  any  (discrete,  continuous,  or  mixed) 
rv 

F(Q(m))>m  and  Q(F(f))<f.  (A-6) 


^Mood,  A.  M.,  F.  A.  Graybill,  and  D.  C.  Boes.  Introduction  to  the  Theory  of  Statistics.  3rd  edition, 
NY:  McGraw-Hill,  1974. 
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A.2  Some  Useful  Random  Variables 


These  particular  random  variables  play  a  part  in  the  procedures  discussed  and 
developed  in  this  report.  In  general,  A  ~  B  means  that  the  random  variables  A 
and  B  have  the  same  distribution.  So  can  be  read  "has  the  same  distribution 
as"  or  "is  distributed  as"  or  "is  equal  in  distribution  to." 

A  scalar  Bernoulli(0)  or  binomial  random  variable  Z  is  defined  on  S 

=  {0,1},  and  the  parameter  is  0  =  Pr[Z  =  1]  =  1  —  Pr[Z  =  0].  So  its  probability 
function  is 

/0(t)=Pre[Z  =  f]  =  e^(')(')(l-0/(o)''^  te{0,l},  (A-7) 

and  its  moments  are 

EZ  =  e  and  VarZ  =  e(l-e).  (A-8) 

A  sample  of  n  independent  B(1,0)  random  variables  has  the  usual  binomial 
distribution  B{n,6).  In  the  vector  case,  the  binomial  B^^(l,e)  random  variable  is 
Z  =  (Z,)JLi  with  parameter  0  =  Each  component  Z,-  ~  B(1,0,),  and  they 

are  independent.  So  Z  is  defined  on  S  =  {0, 1 }  x  •  •  •  x  (0, 1 }  =  {0, 1 }  =  {0, 1  }* . 

The  probability  function  of  Z  is 

k  k 

fe{t)  =  ?Te[Z  =  t]  =  nPre[Z,-  =  t,]  =  (A-9) 

i=l  1=1 

where  t  =  (ti)/=i  €  {0,1}*^.  Moments  are  given  by 

EZi  =  0i  and  Var  Z,- =  0,(1  -  0,) .  (A-10) 

A  sample  of  n  independent  Bjt(l,0)  random  variables  has  the  vector  binomial 
distribution  Bjt(n,0). 

A  scalar  multinomial  M(l,2,0)  random  variable  Z  has  parameter  {z,d)  where  z 
=  {z;)/=i  and  e  =  (0;)"=i  with  1  and  0  <  0;  <  1  for  each  ;.  The 

distribution  is  given  by  Pr[Z  =  Zj]  =  6j,  so  the  sample  space  is  S  =  {2/}"=i.  The 
probability  function  for  Z  is 

/.,.(»)  =  Pr,,«|Z  =  (l  =  ne''‘'’''’.  te{2, (A-ll) 

;=i 

and  its  moments  are  E  Z”'  = 

n  n 

EZ  =  ^0;Zy  and  VarZ  =  ^0yzj- (EZ)^  (A-12) 

;=1  /=i 
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A  sample  of  n  independent  M(l,2,0)  random  variables  has  the  usual  multinomial 
distribution  M{n,z,e).  The  vector  multinomial  Mk{l,z,e)  random  variable  Z 
=  (Z,)|Li  has  parameter  {z,  6),  where  z  =  {zi%i  and  6  =  (e,)f=i-  The  component 
Zi  ~  M{l,ZiA),  where  z;  =  and  0/  =  The  sample  space  is 

S  =  (S>-Li{z,j}”^i.  The  probability  ftmction  is 


(A-13) 


where  t  =  (tj ) -Lj  e  and  moments  are  given  by  E  Zf  =  X)”ii  so 

n,  «i 

EZi  =  J^ei,jZi,j  and  Var  Z,- =  J]) 0,-,yz?y  -  (E Z,-)^  (A-14) 

;=i  /=i 

In  fact,  Bk{l,e)  ~  Mjt(l,(0,l),(l  -  0,0)).  A  sample  of  n  independent  M*;(l,z,0) 
random  variables  has  the  vector  multinomial  distribution  Mjt(n,z,  0). 


A.3  Hypothesis  Testing 

A  hypothesis  test  is  a  procedure  for  determining  the  truth  of  some  conjecture  about 
the  parameter  of  a  probability  distribution  based  on  data,  which  is  an  observation 
from  the  distribution.  Suppose  the  data  U  is  a  random  variable  defined  on  a 
sample  space  S  according  to  a  probability  law  with  parameter  0,  and  that  T :  S  — *•  R 
is  a  function  so  that  T{U)  is  a  statistic.  Consider  the  h5^othesis  test 

Ho  :  0  =  y 

(A-15) 

which  asserts  that  the  correct  value  of  0  is  in  fact  y.  If  small  T  indicates  that  0  =  y 
and  large  T  indicates  otherwise,  then  T  is  said  to  have  large  values  significant.  The 
decision  rule  is  to  reject  Ho  if  T  >  where  the  critical  level  Ta  is  chosen  so  that 
the  probability  of  an  incorrect  decision  under  the  null  hypothesis,  or  Type  I  error, 
is  no  greater  than  Up.  This  requires  that  Ta  is  large  enough  that  Pr^fT  >  Ta]  <  a© 
but  no  larger.  So  choose 


Ta  =  inf{T  :  Pry[T  >  !«]  ^  Uo} 

=  inf{r  :  Pry[T  ^  ^  1  -  ap] 

=  inf{T  ;  Fy(t}  ^  1  -  ap) 

=  Qy{l-ap).  (A-16) 
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Then  the  actual  probability  of  Type  I  error  (also  called  the  size  of  the  critical  region 
or  significance  level  of  the  test)  is 


a  =  Pry  [T  >Ta]  =  l-  Pry[T  ^  T^]  =  1  -  Fy(Ta)  =  1  -  Fy((2y  (1  -  «„)) 

<  1  -  (1  -  tto)  =  tto  (A-17) 

as  required.  The  p-value  of  the  test,  denoted  p,  is  defined  as 

p  =  l-Fy{T).  (A-18) 

If  p  <  a,  then  fy(T)  >  Fy(Ta)  and  T  >  Ta.  Thus,  p  <  a  implies  that  Hq  is  to  be 
rejected. 

The  Type  II  error  probability  ^  of  making  an  incorrect  decision  under  an  alternative 
with  the  specific  parameter  value  6  is 


i3  =  Pre[r<T«]  =  Fe(T„), 


(A-19) 


and  the  probability  of  making  the  correct  decision  under  9  is  the  power  of  the  test 

l-j8  =  Pre[r>T„]  =  l-Fe(T„).  (A-20) 


(X(,))jLj  denotes  the  order  statistics  of  so  that  X(,)  ^  X(,+i).  And 

denotes  rank,  so  X(,j  =  i.  Descending  rank  will  be  denoted  by  X~,  so 
X^)  =  n  + 1  -  i. 
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